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Abstract — In this note, new Hajj and Umrah Event Recognition 
datasets (HUER) are presented. The demonstrated datasets are 
based on videos and images taken during 2011-2012 Hajj and 
Umrah seasons. HUER is the first collection of datasets covering 
the six types of Hajj and Umrah ritual events (rotating in 
Tawaf around Kabaa, performing Sa'y between Safa and Marwa, 
standing on the mount of Arafat, staying overnight in Muzdalifah, 
staying two or three days in Mina, and throwing Jamarat). 
The HUER datasets also contain video and image databases for 
nine types of human actions during Hajj and Umrah (walking, 
drinking from Zamzam water, sleeping, smiling, eating, praying, 
sitting, shaving hairs and ablutions, reading the holy Quran and 
making duaa). The spatial resolutions are 1280x720 pixels for 
images and 640x480 pixels for videos and have lengths of 20 
seconds in average with 30 frame per second ratesQ 



I. Introduction 

In the last decade the field of visual recognition had an 
outstanding evolution from classifying instances of objects 
towards recognizing the classes of objects and scenes in 
natural images. Much of this progress has been sparked by 
the creation of realistic image datasets as well as by the new 
and robust methods for image description and classification. 
We take inspiration from this progress and aim to transfer 
previous experience to the domain of video recognition and 
the recognition of human actions in particular during Hajj and 
Umrah seasons (4). 

Action recognition from video shares common problems 
with object recognition in static images. Both tasks have to 
deal with significant intra-class variations, background clutter 
and occlusions. In the context of object recognition in static 
images, these problems are surprisingly well handled by a bag- 
of-features representation combined with state-of-the-art 
machine learning techniques like support vector machines. It 
remains, however, an open question whether and how these 
results generalize to the recognition of realistic human actions, 
e.g., in feature films or personal videos. 

The Hajj and Umrah event recognition datasets are capable 
of recognizing a wide range of human actions during the Hajj 
and Umrah rituals (Tawaf, Sa'y, Arafat, etc) under different 
conditions. The main goal is to develop the Hajj and Umrah 
event recognition datasets to solve all the following problems: 

• Detecting injured, dead, and sleeping pilgrims inside 
Masjid El-Harram and around Kabaa. 
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Fig. 1. Hajj and Umrah Events Recognition Datasets. Various images are 
taken from different places representing Hajj and Umrah rituals. 



Detecting abnormal pilgrims during Safa/Marwa and rec- 
ognizing overcrowded areas. 

Detecting abnormal human events in Makkah and in 
particular outside El-Harram. 

Detecting people setting in sidewalks, allies, sub-roads, 
and stairs. 

Detecting missing and found people or objects inside El- 
Harram and outside El-Harram. 

Detecting and recognizing empty spaces inside and out- 
side El-Harram. 



II. Hajj and Umrah rites activities and duties 

The Hajj is the fifth pillar of Islam and the Hajj is one of 
the largest pilgrimages in the world. The Hajj and Umrah are 
a demonstration of the solidarity of the Muslim people, and 
their submission to Allah. Many pilgrims come simultaneously 
converge on Makkah to do Hajj and Umrah and perform a 
series of rituals: Each pilgrim walks seven times around the 
Kaaba counter-clockwise, then runs back and forth between 




Fig. 2. All Hajj and Umrah ritual locations 

the Al-Safa and Al-Marwah hills, also the pilgrims always 
drink from the Zamzam Well, however the pilgrims go to the 
plains of mount of Arafat to stand in vigil, and throws stones in 
a ritual Stoning of the Devil, see for example . The pilgrims 
then shave their heads, perform a ritual of animal sacrifice, and 
celebrate the three day global festival of Eid al-Adha, see the 
Hajj and Umrah events' recognition in Fig. [T] 

There are six hajj and umrah ritual events, which are 
modeling during Hajj and Umrah as seen in Fig. [2] These 
are circling in Tawaf, performing Sa'y between Safa and 
Marwa, standing on the mount of Arafat, staying overnight 
in Muzdalifah, staying three nights in Mina, and throwing 
Jamarat Q. The models defined for this study are described 
below: 

1) Pilgrims Tawaf: 

The pilgrims Tawaf around Kaaba is one of the most 
Islamic rituals. During the Hajj and Umrah, Muslims have 
to circumambulate the Kaaba (the most sacred site in 
Islam, and the most sacred place on earth) seven times, 
in a counterclockwise direction J6] 0, see Fig. [3] 




Fig. 3. Examples of the pilgrims circling in Tawaf 

2) Sa'y between Safa and Marwa: Pilgrims, whether they 
are performing Hajj or Umrah, perform Sa'y after Tawaf. 
Sa'y means endeavoring or making effort. For Hajj, this 
is held to commemorate Hagar's running between Safa 
and Marwa seven times in order to find water for her 
son, Ishmael, whom she was still breast-feeding [6) Q, 
see Fig. 01 

3) Standing on the mount of Arafat: The plain and 
mount of Arafat are located in the south-east side of 




Fig. 4. Examples of Sa'y between Safa and Marwa 



Makkah approximately 12 kilometers away of El-Harram, 
Saudi Arabia, about three million pilgrims congregated 
to perform the most important rite of the Hajj, or the 
pilgrimage. This rite is significant because it is on the 
mount of Mercy that the Prophet Muhammad gave his fi- 
nal sermon. Many pilgrims climb the hill and try to touch 
the pillar that marks this place. After Arafat, pilgrims will 
move to Muzdalifah to complete the remaining rites of 
the pilgrimage (6) Q, see Fig. [5] 




Fig. 5. Examples of standing on the mount of Arafat 

4) Staying overnight in Muzdalifah: Staying in Muzdal- 
ifah is obligatory upon the one performing the Hajj to 
spend the tenth (10 th ) of Dhul-Hijjah until the time of 
Fajr prayer Q, see Fig. 




Fig. 6. Examples of staying overnight in Muzdalifah 

5) Staying overnight in Mina: Mina, seven kilometers east- 
south of the Masjid El-Harram is where Hajj pilgrims 
sleep overnight on the 8 th , 11 th , 12 th (and some even 
on the 13 th ) of Dhul Hijjah. It contains the Jamarat, the 
three stone pillars which are pelted by pilgrims as part 
of the rituals of Hajj O Q, see Fig. □ 

6) Throwing Jamarat: The pilgrim who throws Jamarat 
before Zawal on the 11 th day and the following days will 
have to throw the pebbles again after Zawal if the days 
of throwing the pebbles have not yet expired Q, see 
Fig.S 




Fig. 7. Examples of staying overnight in Mina 




Fig. 8. Examples of throwing Jamarat 

III. Hajj and Umrah Pilgrim Event Classifications 

Nine pilgrim events will be modeled during Hajj and Umrah 
rituals, as shown in Fig. [9] They are drinking from Zamzam 
water, walking pilgrims, smiling pilgrims, sleeping pilgrims, 
sitting pilgrims, eating pilgrims, praying pilgrims, shaving 
hairs and ablution pilgrims, and reading quran and making 
duaa. The models defined for this study are described below: 



1 2 3 4 5 6 




Fig. 9. Hajj and Umrah Pilgrim Events Recognition Datasets. Various images 
are taken from different places representing Hajj and Umrah rituals. 

1) Walking pilgrims: Walking during Hajj corresponds to 
a significant number of pilgrims moving at a low speed. 
The pilgrims are walking in all rites for long distances 
in huge numbers, which resulting in overcrowding and 
suffocation, see Fig. [TUJ 

2) Drinking from Zamzam water: The Zamzam well is 
located in Makkah, which is the heart of the Hajj pilgrim- 




Fig. 10. Examples of the walking pilgrims 

age. Standing only a few meters east of the Kabaah, the 
well is 35 meters deep and topped by an elegant dome. In 
last decades, scientists have collected samples of Zamzam 
water and they have found certain peculiarities that make 
water healthier, like a higher level of calcium, see Fig. Qj] 




Fig. 11. Examples of drinking from Zamzam water 

3) Sleeping pilgrims: This event will recognize the Pilgrims 
sleeping from severity of fatigue and exhaustion during 
Hajj and Umrah rites. Therefore, some of them may resort 
to sleep in the streets, putting their lives at risk, see 
Fig. EH 




Fig. 12. Examples of the sleeping pilgrims 

4) Eating pilgrims: This event will recognize the pilgrims 
eating during the Hajj and Umrah in the camps and places 
to spend the night in Mina and Muzdalifah, see Fig. [T3l 




Fig. 13. Examples of the eating pilgrims 

5) Al-Haram Prayer pilgrims: This event will recognize 
pilgrims performing most or all five daily prayers in the 
most beautiful and purest part of the earth in the Holy 
Mosque in Makkah, see Fig. [T4j 
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especially in the month of Ramadan, the pious Muslims 
read regularly the Holy Quran, see Fig. IT8l 



Fig. 14. Examples of the praying pilgrims 



6) Smiling pilgrims: The smiling pilgrims is coming from 
their great happiness because they are in the best place 
where the survival of psychological serenity and peace of 
mind because they are with God (ALLAH), see Fig. [T5l 




Fig. 15. Examples of smiling pilgrims 

7) Sitting pilgrims: 

This event will recognize pilgrims siting during and after 
the performance of rituals, from the severity of fatigue. 
There are many places where having rest is possible, but 
some pilgrims may sit down and resting in the streets, 
which could lead to congestion in some of these areas 
from the intensity of the crowds, see Fig. [16] 




Fig. 16. Examples of the sitting pilgrims 

8) Ablutions and shaving hairs: This event will recognize 
ablution, which is the cornerstone of the prayer and 
can't valid without it, as well as it recognizes pilgrims 
shaving their hair during the Hajj or Umrah, see Fig. [17] 




Fig. 17. Ablutions and shaving hairs 

9) Reading quran and making duaa: Reading the Holy 
Quran is one of the most important worship by Muslims, 




Fig. 18. Reading the holy Quran and making duaa 



IV. Discussion and Conclusion 

Up to our knowledge, the proposed human event recognition 
datasets are the first event recognition datasets to be devel- 
oped to observe and model the Hajj and Umrah activities in 
Makkah. We plan to develop algorithms for event recognition, 
videos extractions, and classifications based on these devel- 
oped datasets. 

Other datasets of human events can be briefly described 
as follows. KTH dataset is one of the known event recog- 
nition datasets for human actions, which introduces a video 
database containing six types of human actions (walking, 
jogging, running, boxing, hand waving, and hand clapping) 
performed by 25 people. The sequences are downsampled to 
the spatial resolution of 160x120 pixels and have a length 
of four seconds in average with 25 fps frame rate Q. 
The Weizmann dataset is another human action dataset, it 
contains a database of 90 low-resolution (180x144 pixels) 
with 50 fps frame rate. The video sequences performed by 9 
people, each one performing 10 natural actions such as (run, 
walk, skip, jumping-jack, jump-forward-on-two-legs, jump- 
in-place-on-two-legs, gallopsideways (side), wave-two-hands, 
waveone-hand, and bend) l5l . 
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